Skip to content

feat(analytics): add autocapture frustration signals (click, rage click, dead click)#982

Open
rahul-mixpanel wants to merge 66 commits into
masterfrom
feature/autocapture-phase1
Open

feat(analytics): add autocapture frustration signals (click, rage click, dead click)#982
rahul-mixpanel wants to merge 66 commits into
masterfrom
feature/autocapture-phase1

Conversation

@rahul-mixpanel

@rahul-mixpanel rahul-mixpanel commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Implements autocapture Phase 1: $mp_click, $mp_rage_click, and $mp_dead_click event detection for both XML Views and Jetpack Compose UIs
  • Adds AutocaptureOptions configuration (disabled by default, opt-in via MixpanelOptions builder) with per-feature options (ClickOptions, RageClickOptions, DeadClickOptions)
  • Intercepts touch events via Window.Callback wrapping, with WindowSpy for dialog/popup/menu window tracking
  • Full Compose support via native SemanticsNode tree traversal (graceful degradation when Compose is not present)

Privacy

Autocapture does not capture visible text content ($el_text) from tapped elements. Tracking text can be invasive and raise privacy concerns. The complexity of nested view hierarchies can also cause text extraction to capture content from unintended views — for example, tapping a LinearLayout container might extract text from a deeply nested TextView that isn't semantically related to the tap.

The captured properties ($el_id, $el_tag_name, $attr-aria-label, $attr-role, $elements, $x, $y) are purely structural UI metadata with no PII risk.

Captured Properties

Property Description
$x, $y Touch coordinates
$el_id Element identifier (contentDescription → resource ID → fallback hash)
$el_tag_name Element class name (e.g., Button, TextView)
$attr-aria-label Accessibility label (contentDescription)
$attr-role Semantic role (e.g., Button, Switch, Slider)
$elements View hierarchy string (max 5 levels)

Architecture

Component Responsibility
TouchInterceptor Per-window touch interception via Window.Callback
SemanticExtractor Element info extraction (XML View + Compose routing)
ComposeSemanticHelper Compose-specific semantics via RootForTest / SemanticsNode
RageClickTracker 4+ clicks within 1000ms / 44dp radius
DeadClickDetector No UI change within 500ms (XML: view hash, Compose: semantic snapshot)
AutocaptureManager Lifecycle coordination, event dispatch
WindowSpy Root view tracking for dialogs, popups, menus

Tests

  • 7 XML autocapture instrumented tests (AutocaptureInstrumentedTest)
  • 7 Compose autocapture instrumented tests (ComposeAutocaptureInstrumentedTest)
  • All 14 autocapture tests pass on emulator (API 35)

Demo app

  • XML and Compose test screens added to mixpaneldemo for manual validation

Hardening (code review fixes)

  • Depth-limited recursion (max 20) on all tree traversals to prevent StackOverflowError
  • Fixed WeakHashMap iteration in stop() to avoid ConcurrentModificationException
  • Fixed WindowSpy.getRootViews() returning stale data
  • Fixed ClickEvent.isComposeClick() unreliability from WeakReference GC
  • Fixed AccessibilityNodeInfo leak on exception

Test plan

  • Run XML tests: ./gradlew :analytics:connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.mixpanel.android.mpmetrics.AutocaptureInstrumentedTest
  • Run Compose tests: ./gradlew :analytics:connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.mixpanel.android.mpmetrics.ComposeAutocaptureInstrumentedTest
  • Manual validation using demo app XML and Compose test screens
  • Verify existing SDK tests still pass: ./gradlew :analytics:connectedAndroidTest

rahul-mixpanel and others added 13 commits June 11, 2026 23:33
…ead click

Add autocapture functionality for automatic event tracking:

- Click tracking ($mp_click): Captures all touch events with semantic info
- Rage click detection ($mp_rage_click): 4+ clicks within 1000ms in 44dp radius
- Dead click detection ($mp_dead_click): No UI change within 500ms after click

Key components:
- AutocaptureManager: Main coordinator with activity lifecycle integration
- TouchInterceptor: Window.Callback wrapper for touch interception
- WindowSpy: Tracks all windows (dialogs, popups) via reflection
- SemanticExtractor: Extracts element info from View hierarchy
- RageClickTracker: Click pattern detection with spatial/temporal thresholds
- DeadClickDetector: UI change monitoring with ViewTreeObserver

Configuration via MixpanelOptions.Builder.autocaptureOptions()

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add test screens for validating autocapture functionality:

- ComposeAutocaptureTestScreen: Jetpack Compose test screen
- XmlAutocaptureTestActivity: XML Views test screen
- DemoApplication: Enables autocapture during SDK initialization

Test coverage includes:
- $el_id resolution (contentDescription, resource ID, hash fallback)
- Dead click detection (elements with no UI response)
- Rage click detection (4+ rapid taps)
- Excluded controls (Switch, EditText, SeekBar)
- Privacy filtering (mp-sensitive, mp-no-track)

Note: XML views autocapture working; Compose needs improvement.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…erent feedback

Privacy filtering:
- Return null (block all events) for views marked mp-sensitive or mp-no-track
- Add same check for Compose via AccessibilityNodeInfo
- Add debug logging when skipping sensitive elements

Dead click exclusions:
- Exclude EditText (keyboard appears, cursor shows)
- Exclude CompoundButton (Switch, CheckBox, RadioButton - toggle animation)
- Exclude SeekBar (thumb moves with drag)

These controls have inherent visual feedback and should not trigger $mp_dead_click.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ose APIs

- Add ComposeSemanticHelper to extract semantics from Jetpack Compose views
  using Compose's SemanticsNode API directly (no reflection)
- Add compileOnly dependency on Compose UI for graceful degradation
- Update SemanticExtractor to detect Compose roots and use ComposeSemanticHelper
- Implement $el_id resolution: contentDescription > testTag > hash fallback
- Implement sensitive element detection (mp-sensitive/mp-no-track) with
  ancestor checking - returns SENSITIVE_BLOCKED to prevent accessibility fallback
- Disable dead click detection for Compose (ViewTreeObserver cannot detect
  Compose UI changes) - click and rage click work correctly

Note: Dead click for Compose requires semantic tree comparison (future enhancement)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reorder property checks so EditableText is checked before OnClick.
TextFields in Compose are clickable (to gain focus) but should be
identified as TextField/textbox, not Button/button.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ViewTreeObserver cannot detect Compose UI changes since Compose renders
inside a single AndroidComposeView. This implements semantic tree comparison
for Compose clicks: captures semantic hash at click position (text, states,
bounds, children), then compares after timeout to detect UI changes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…snapshot comparison

Use semantic tree snapshot comparison for Compose dead click detection:
- Capture snapshot immediately at click (nodeCount + contentHash)
- At timeout, detect changes via content hash or significant node count change
- Navigation detected by node count increase (new screen composables added)
- Ripples don't affect semantic tree, so dead clicks are correctly detected

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add complete integration guide for autocapture feature covering click,
rage click, and dead click tracking.

- Add analytics/docs/autocapture.md (620 lines)
  - Overview and quick start with Kotlin/Java examples
  - Configuration options for all event types
  - Complete event properties reference
  - Element identification best practices
  - Full Jetpack Compose support documentation
  - Multi-window tracking explanation (WindowSpy)
  - Privacy considerations and sensitive data handling
  - Troubleshooting guide with debug logging
  - ProGuard configuration notes
  - Migration guide from manual tracking
  - FAQ section

- Update analytics/README.md
  - Add Features section with Autocapture overview
  - Add quick setup examples (Kotlin + Java)
  - Link to detailed autocapture guide
  - Update table of contents

Documentation provides comprehensive coverage exceeding iOS guide,
with dual-language examples and complete API reference.
Add 9 instrumented tests covering:
- Click event capture and property verification
- Element ID resolution (contentDescription, resource ID, hash fallback)
- Rage click detection with rapid touch injection
- Dead click detection for unresponsive elements
- Privacy filtering for mp-sensitive elements
- Multiple click event generation
- Token and distinct_id verification
- Add testElementIdResolutionRule3HashFallback: verifies hash fallback
  format (Button_view_<hex>) when both contentDescription and resource
  ID are absent
- Add testClickEventCapturesElText: verifies $el_text captures the
  button's visible text content via SemanticExtractor.extractText
Add 9 instrumented tests for Jetpack Compose autocapture, mirroring
the XML test suite:
- Click event capture with contentDescription element ID
- Element ID from testTag (Compose Rule 2 fallback)
- Element ID hash fallback (no contentDescription, no testTag)
- Rage click detection with rapid sendPointerSync injection
- Dead click detection (onClick={} with no UI change)
- Privacy filtering for mp-sensitive composables
- Multiple click event generation
- $el_text capture from Text composable
- Token and distinct_id verification

Uses sendPointerSync for touch injection because compose-ui-test's
performTouchInput bypasses Window.Callback where TouchInterceptor
lives. Compose test rule is used only to find element bounds.
Remove testClickEventPropertiesComplete and testElementIdResolutionRule1
as they duplicate coverage already provided by testXmlClickEventBasic.
- Add MAX_RECURSION_DEPTH (20) to prevent StackOverflowError in
  countViews, computeContentHash, computeTreeHash,
  findNodeAtPositionRecursive, and collectTextFromChildren
- Fix WeakHashMap iteration in stop() to avoid ConcurrentModificationException
- Fix WindowSpy.getRootViews() returning stale data by pointing to the
  live delegating list instead of the replaced original
- Fix ClickEvent.isComposeClick() unreliability by using a boolean flag
  set at construction time instead of checking WeakReference at read time
- Fix AccessibilityNodeInfo leak on exception in findNodeAtPosition
@rahul-mixpanel
rahul-mixpanel requested review from a team and tylerjroach June 29, 2026 16:06
@rahul-mixpanel rahul-mixpanel changed the title feat(autocapture): implement Phase 1 - click, rage click, dead click feat(analytics): add frustration signals - click, rage click, dead click Jun 29, 2026
@rahul-mixpanel rahul-mixpanel changed the title feat(analytics): add frustration signals - click, rage click, dead click feat(analytics): add autocapture frustration signals (click, rage click, dead click) Jun 29, 2026
@rahul-mixpanel
rahul-mixpanel marked this pull request as draft June 29, 2026 16:36
@rahul-mixpanel rahul-mixpanel self-assigned this Jun 29, 2026

@tylerjroach tylerjroach left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't finished reviewing the code, but had these questions on autocapture doc

Comment thread analytics/docs/autocapture.md Outdated
Comment thread analytics/docs/autocapture.md
Comment thread analytics/docs/autocapture.md
Comment thread analytics/docs/autocapture.md Outdated
Comment thread analytics/docs/autocapture.md
Comment thread analytics/docs/autocapture.md Outdated
Comment thread analytics/README.md Outdated
…tent

$el_text (visible text content of tapped elements) is no longer collected
by default. Users must explicitly enable it by setting captureTextContent
to true in AutocaptureOptions, protecting user privacy by default.
Simplify element exclusion to a single marker. Both mp-sensitive and
mp-no-track had identical behavior (full block), so only mp-no-track
is needed.
Comment thread analytics/src/main/java/com/mixpanel/android/mpmetrics/MixpanelOptions.java Outdated
Comment thread analytics/src/main/java/com/mixpanel/android/autocapture/WindowSpy.java Outdated
Comment thread analytics/src/main/java/com/mixpanel/android/autocapture/WindowSpy.java Outdated
Comment thread analytics/src/main/java/com/mixpanel/android/autocapture/TouchInterceptor.java Outdated
…xclusion

Remove text content capture entirely from autocapture. Tracking visible
text can be invasive and raise privacy concerns. The complexity of nested
view hierarchies can also cause text extraction to capture content from
unintended views — for example, tapping a container layout might extract
text from a deeply nested label unrelated to the tap.

Without text capture, the mp-no-track element exclusion marker is no
longer needed since the remaining properties ($el_id, $el_tag_name,
$attr-aria-label, $attr-role, $elements) are purely structural UI
metadata with no PII risk.

Removed: captureTextContent config, $el_text property, text extraction
and sanitization (CC/SSN regex), sensitive input detection, mp-no-track
marker checks, and all related tests and demo elements.
@greptile-apps

greptile-apps Bot commented Jul 1, 2026

Copy link
Copy Markdown

Confidence Score: 3/5

Hold for merge until the two confirmed missing event properties are fixed — every Compose click event will be missing $elements, and $attr-aria-label is dropped in the accessibility fallback path when a resource ID exists.

The hardening round successfully addressed multi-touch false taps, eventTime float precision, AccessibilityNodeInfo leaks, recursion depth limits, and null guards. However, $elements is entirely absent from all direct Compose SemanticsNode events and $attr-aria-label is dropped in the accessibility path when viewIdResourceName is found first — both confirmed present in the current code and affecting every Compose click event the SDK tracks.

ComposeSemanticHelper.java (extractFromNode never calls builder.elements()) and SemanticExtractor.java (extractFromNode accessibility path drops contentDescription once viewIdResourceName is set).

Important Files Changed

Filename Overview
analytics/src/main/java/com/mixpanel/android/autocapture/AutocaptureManager.java Core lifecycle coordinator; stop() now wired to optOutTracking(). Windowless-view touch listeners are no longer installed (only windows get listeners), resolving the previous cleanup gap.
analytics/src/main/java/com/mixpanel/android/autocapture/CurtainsHelper.kt ACTION_POINTER_DOWN and ACTION_CANCEL now reset downTime, fixing the multi-touch false-tap. eventTime stored as Long (not float), fixing the precision regression.
analytics/src/main/java/com/mixpanel/android/autocapture/SemanticExtractor.java findViewAtPosition now depth-limited. AccessibilityNodeInfo recycled in finally block. $attr-aria-label still silently dropped in the accessibility path when viewIdResourceName is found first.
analytics/src/main/java/com/mixpanel/android/autocapture/ComposeSemanticHelper.java Direct SemanticsNode extraction path still omits builder.elements(), so every Compose-originated event will be missing $elements.
analytics/src/main/java/com/mixpanel/android/autocapture/DeadClickDetector.java Unified XML/Compose detection strategy is well-structured. XOR accumulation for mixed-framework content hash is intentional and documented. DetectionSession cleanup is correct.
analytics/src/main/java/com/mixpanel/android/autocapture/RageClickTracker.java Straightforward implementation; time-window pruning and radius check are correct. Clears on activity pause to avoid cross-screen false positives.
analytics/src/main/java/com/mixpanel/android/mpmetrics/MixpanelAPI.java Autocapture initialised in constructor, stopped in optOutTracking(), restarted in optInTracking(). The isAutomaticEvent routing warrants confirming (previous outside-diff concern).
analytics/src/main/java/com/mixpanel/android/mpmetrics/AutocaptureOptions.java Null guards added to all Builder setters; silently ignores null to protect host app.
analytics/src/main/java/com/mixpanel/android/autocapture/WindowSpy.java Curtains-backed root-view tracking; CopyOnWriteArrayList listeners are thread-safe. Each AutocaptureManager listener is removable via removeListener().

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant User
    participant Window
    participant CurtainsHelper
    participant AutocaptureManager
    participant SemanticExtractor
    participant ComposeSemanticHelper
    participant DeadClickDetector
    participant MixpanelAPI

    User->>Window: Touch DOWN
    Window->>CurtainsHelper: touchEventInterceptors (ACTION_DOWN)
    CurtainsHelper->>CurtainsHelper: record downX, downY, downTime
    User->>Window: Touch UP
    Window->>CurtainsHelper: touchEventInterceptors (ACTION_UP)
    CurtainsHelper->>CurtainsHelper: validate tap
    CurtainsHelper->>AutocaptureManager: onTap(x, y, decorView)
    AutocaptureManager->>SemanticExtractor: extract(decorView, x, y)
    alt Compose view
        SemanticExtractor->>ComposeSemanticHelper: extract(composeRoot, screenX, screenY)
        ComposeSemanticHelper-->>SemanticExtractor: ClickEvent.Builder
    else XML view
        SemanticExtractor->>SemanticExtractor: extractFromView
    end
    SemanticExtractor-->>AutocaptureManager: ClickEvent
    AutocaptureManager->>MixpanelAPI: trackClick
    AutocaptureManager->>DeadClickDetector: startDetection
    DeadClickDetector->>DeadClickDetector: postDelayed(500ms)
    alt No UI change
        DeadClickDetector->>MixpanelAPI: trackDeadClick
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant User
    participant Window
    participant CurtainsHelper
    participant AutocaptureManager
    participant SemanticExtractor
    participant ComposeSemanticHelper
    participant DeadClickDetector
    participant MixpanelAPI

    User->>Window: Touch DOWN
    Window->>CurtainsHelper: touchEventInterceptors (ACTION_DOWN)
    CurtainsHelper->>CurtainsHelper: record downX, downY, downTime
    User->>Window: Touch UP
    Window->>CurtainsHelper: touchEventInterceptors (ACTION_UP)
    CurtainsHelper->>CurtainsHelper: validate tap
    CurtainsHelper->>AutocaptureManager: onTap(x, y, decorView)
    AutocaptureManager->>SemanticExtractor: extract(decorView, x, y)
    alt Compose view
        SemanticExtractor->>ComposeSemanticHelper: extract(composeRoot, screenX, screenY)
        ComposeSemanticHelper-->>SemanticExtractor: ClickEvent.Builder
    else XML view
        SemanticExtractor->>SemanticExtractor: extractFromView
    end
    SemanticExtractor-->>AutocaptureManager: ClickEvent
    AutocaptureManager->>MixpanelAPI: trackClick
    AutocaptureManager->>DeadClickDetector: startDetection
    DeadClickDetector->>DeadClickDetector: postDelayed(500ms)
    alt No UI change
        DeadClickDetector->>MixpanelAPI: trackDeadClick
    end
Loading

Reviews (34): Last reviewed commit: "Update tests to match shortened elementI..." | Re-trigger Greptile

Comment thread analytics/src/main/java/com/mixpanel/android/mpmetrics/MixpanelOptions.java Outdated
- Remove redundant API 19 check in WindowSpy (minSdk is 21)
- Guard autocaptureOptions(null) with warning instead of NPE
- Add depth limit to findViewAtPosition for parity with Compose path
- Expand dead click excluded controls docs with full platform breakdown
- Add Javadoc to AutocaptureOptions.Builder clarifying defaults
@tylerjroach

Copy link
Copy Markdown
Contributor

Not to overwhelm here but some notable findings to look into from Fable.

1. Autocapture events silently dropped when trackAutomaticEvents=false

MixpanelAPI.java:2204 — The emit callback calls track(eventName, properties, /*isAutomaticEvent=*/true), and track() (line 2805) early-returns on (isAutomaticEvent && !mTrackAutomaticEvents). An app created with getInstance(context, token, false, optionsWithAutocaptureEnabled) starts the full autocapture machinery (interceptors, WindowSpy, timers) yet emits zero events — "Autocapture initialized" is logged and nothing hints at the drop. Autocapture opt-in should gate itself rather than piggyback on the automatic-events flag.

2. WindowSpy mViews swap is an unsynchronized race that can crash the host app

WindowSpy.java:80-84install() copies WindowManagerGlobal.mViews into a new list and writes the field back while holding only WindowSpy's own lock (not the framework's mLock), running synchronously on whatever thread calls getInstance(). A root view added by the main thread between the copy and the field write vanishes from mViews; its later removeView throws IllegalArgumentException("View not attached to window manager") in the host app — violating the never-crash rule. This swap technique is only safe when performed on the main thread.

3. Every single-pointer ACTION_UP is treated as a click — scrolls and swipes emit $mp_click

TouchInterceptor.java:107-125ACTION_DOWN is never recorded, so there is no touch-slop or duration check, and ACTION_CANCEL/ACTION_MOVE are ignored (no mitigation exists anywhere downstream). Every scroll stroke on a RecyclerView/LazyColumn produces a spurious click on whatever was under the finger at lift-off, and a few quick strokes in one region synthesize a false $mp_rage_click. A slop/duration check at this layer is standard.

4. Responsive buttons falsely reported as dead clicks (XML path)

DeadClickDetector.java:201-224 — The baseline snapshot is taken 150ms after the click, and onGlobalLayout/onScrollChanged explicitly ignore anything before mBaselineCaptured ("expected settling"). A click handler that updates the UI within a frame or two — the common case — has its response absorbed into the baseline; at the 500ms check the tree matches and a false $mp_dead_click fires. The Compose path does this correctly by snapshotting synchronously at click time.

5. No retroactive attach — deferred SDK init captures nothing on the current screen

AutocaptureManager.java:101-122start() only registers forward-looking hooks: registerActivityLifecycleCallbacks doesn't replay onActivityResumed for an already-resumed activity, and DelegatingViewList's copy-constructor bulk copy bypasses the overridden add(), so existing root views never fire onRootViewChanged. With deferred init (e.g., after a consent dialog), the foreground activity gets no interceptor until navigation or a config change.

6. Dialogs, popups, and menus are never captured — WindowSpy attaches to nothing

AutocaptureManager.java:339-357getWindowFromView() only handles context instanceof Activity directly (no ContextWrapper unwrapping; the "reflection fallback" comment at line 341 has no code behind it). Dialog decor views carry a DecorContext/ContextThemeWrapper, so the method returns null and no interceptor attaches. When the context is the Activity (popups), it returns the activity's already-registered window, so the containsKey guard skips attachment. Net effect: the entire WindowSpy mechanism currently attaches interceptors to nothing.

7. Compose hit-testing mixes coordinate spaces

ComposeSemanticHelper.java:226TouchInterceptor captures getRawX()/getRawY() (screen space) and passes them unconverted to findNodeAtPositionRecursive, which tests against SemanticsNode.getBoundsInWindow() (window space). The XML and accessibility paths correctly use screen space (getLocationOnScreen, getBoundsInScreen). For any Compose window not at the screen origin — dialogs, popups, multi-window — the tap misses or attributes to the wrong node.

8. Two Window.Callback methods are not delegated

TouchInterceptor.java:128-281onProvideKeyboardShortcuts (API 24) and onPointerCaptureChanged (API 26) are not overridden. Both are default no-ops in the interface, so this compiles silently, but once wrapped those framework notifications never reach the host Activity — breaking the keyboard-shortcuts sheet (Meta+/) and pointer-capture notifications. The existing Api23Helper + animalsniffer-ignore pattern covers this cleanly.

9 $tap_count documented but never emitted

RageClickTracker.java:92-95autocapture.md:150 documents $tap_count ("Number of taps in the rage click sequence") on $mp_rage_click, but recordClick clears the click history and returns the triggering event unchanged, and ClickEvent.toProperties() has no tap-count field. The documented property never appears in any emitted event, and once cleared the actual count is unrecoverable.

10. stop() / uninstall() latent defects

TouchInterceptor.java:67-76uninstall() only restores the callback when getCallback() == this; if another SDK wrapped the callback after Mixpanel, it silently no-ops and the stale interceptor keeps emitting clicks forever — there is no detached/stopped flag anywhere on the event path. install() is also not idempotent (no check for an
already-installed TouchInterceptor), so a stop/start cycle would double-wrap and double-emit every click. Currently unreachable — nothing in the SDK calls the public stop() — but worth fixing before it becomes reachable.

Autocapture PR Review — Additional Findings (design, performance, cleanup)

Design / architecture

D1. "Disabled by default" is a hand-built template, not a real off switch

MixpanelOptions.java:147-152 — The default is implemented by building an AutocaptureOptions that individually disables each of the three current sub-options, and AutocaptureOptions.isEnabled() is an OR over sub-options. When Phase 2 adds a new sub-option
whose builder defaults to enabled=true (the existing convention in ClickOptions/RageClickOptions/DeadClickOptions), every app that never called autocaptureOptions() silently gets that capture enabled. Fix: a master enabled flag or an
AutocaptureOptions.disabled() factory so "off" is defined once, in the class that owns the sub-options. (flagged independently by 2 agents)

D2. Dead-click detection is two divergent mechanisms instead of one abstraction

DeadClickDetector.java:188DetectionSession branches on mIsComposeClick throughout start/captureBaseline/checkResult/cleanup, and the paths already disagree: baselineDelayMs silently doesn't apply to Compose, layout/scroll cancellation signals only exist
for XML, and onWindowFocusChanged() (line 124) is never called by anyone — dead code. Every new change signal or Phase 2 surface must be added to both branches. Fix: a UI-change-monitor interface (captureBaseline / hasChanged / attach / detach) chosen once per
click.

D3. Compose change-detection sensitivity is a hardcoded heuristic

ComposeSemanticHelper.java:136 — The Compose path classifies a click as dead unless the node count changes by more than ±5, bypassing both AutocaptureDefaults and DeadClickOptions, while the XML path uses exact count/hash equality. A Compose click that
adds/removes 1-4 nodes (toggling an icon, revealing a badge) is a false dead click, and there's no option to tune it. Sensitivity should be defined once and honored by both snapshot implementations.

D4. MAX_ACCESSIBILITY_NODES doesn't enforce its documented contract

SemanticExtractor.java:215 — The constant (500, documented as "maximum number of accessibility nodes to probe") is actually used as a recursion depth bound in findNodeAtPosition. A broad-but-shallow tree probes unbounded nodes per tap, while 500 as a depth
limit is far too deep to prevent the StackOverflowError it presumably targets (other traversals cap depth at 20). Fix: a shared visited-node counter for total work, with depth reusing MAX_RECURSION_DEPTH.

D5. WindowSpy install is irreversible and failure is silent-permanent

WindowSpy.java:118 — There is no uninstall path (stop() removes listeners but the swapped list stays forever), sInstalled is set true even when the reflection fails, so degradation is permanent and undetectable by the caller, and DelegatingViewList only
intercepts add/remove — not addAll/clear/set — so coverage depends on OS internals. If another library uses the same mViews-swap technique, whichever installs second wraps the other and neither can be removed. Fix: keep the original list for restore-on-stop,
and surface install failure.

D6. Duplicate ActivityLifecycleCallbacks registration

AutocaptureManager.java:109 — The SDK already registers MixpanelActivityLifecycleCallbacks on the same Application (MixpanelAPI.java:2178); AutocaptureManager registers a second, and with multiple Mixpanel instances each adds another. Dispatching
autocapture's resume/pause/destroy hooks from the existing callbacks would keep one lifecycle path to maintain.

D7. XML hit-testing picks the deepest view, not the clickable target

SemanticExtractor.java:506findViewAtPosition returns the deepest visible view with no preference for clickable views or walk-up to a clickable ancestor (unlike the accessibility path). A tap on a clickable container built as ViewGroup + TextView is
attributed to the inner TextView: wrong $el_id, role text instead of button, and isInteractive=false — which also silently skips dead-click detection for that element.

Performance (tap hot path — all on the main thread)

P1. Extraction runs before the app receives the touch event

TouchInterceptor.java:90 — On every ACTION_UP, the full hit test, hierarchy string build, and (for Compose) semantics traversal + snapshot run synchronously before mOriginalCallback.dispatchTouchEvent(event), adding latency between finger-up and the app's

Performance (tap hot path — all on the main thread)

P1. Extraction runs before the app receives the touch event

TouchInterceptor.java:90 — On every ACTION_UP, the full hit test, hierarchy string build, and (for Compose) semantics traversal + snapshot run synchronously before mOriginalCallback.dispatchTouchEvent(event), adding latency between finger-up and the app's click handling. Nothing in the extraction depends on running first — forward the event,
then extract (or post it).

P2. Per-node getLocationOnScreen during hit-test DFS

SemanticExtractor.java:483 — Each visited view allocates an int[2] and calls getLocationOnScreen (which walks the parent chain), making the hit test O(visited × depth) per tap. Converting the tap to root coordinates once and translating incrementally while descending (as ViewGroup.dispatchTouchEvent does) eliminates both costs.

P3. Dead-click snapshots walk the tree twice, twice

DeadClickDetector.java:250captureBaseline() and checkResult() each traverse the full hierarchy twice (countViews + computeContentHash) — 4 full traversals per interactive click where 1-2 suffice. Compute count and hash in a single pass (as ComposeSemanticHelper.computeTreeHash already does), or drop countViews since the hash
already reflects count changes.

P4. Per-node allocation in Compose snapshots

ComposeSemanticHelper.java:207computeTreeHash allocates an int[2] per semantics node, and the snapshot runs at click time and again at timeout — hundreds of short-lived arrays per interactive click on large screens. Pack count+hash into a single long return, or use one holder per snapshot.

P5. Hot-path debug logs build strings regardless of log level

SemanticExtractor.java:60 (also ComposeSemanticHelper.extract, captureSnapshot, AutocaptureManager.emitEvent) — MPLog.d checks the level inside the call, so concatenation and getSimpleName() run on every tap even at the default WARN level. Guard with a level check or remove per-tap logs.

Dead code / simplification

  • AutocaptureManager.java:66mCurrentActivityRef is write-only: assigned every onActivityResumed, never read. Delete. (flagged independently by 3 agents)
  • WindowSpy.java:128getRootViews() and the sOriginalViews field exist only to serve it; zero callers. Delete both. (flagged independently by 2 agents)
  • ClickEvent.java:49timestamp is set, threaded through the 10-arg constructor, and never serialized or read (RageClickTracker keeps its own timestamps). Drop it.
  • DeadClickDetector.java:167DetectionSession caches mIsComposeClick/mComposeRootRef duplicating state already available from the mClickEvent it holds (double-wrapping an already-weak reference). Use the ClickEvent accessors.
  • ComposeSemanticHelper.java:46ExtractResult wrapper + two-value ExtractionResult enum encode exactly "nullable builder". Return @Nullable ClickEvent.Builder and delete both types.
  • SemanticExtractor.java:158,183,293 — Dead SDK-version guards for API 16/18; minSdk is 21, so the early-return is unreachable and the gates are always true.
  • SemanticExtractor.java:516 / ComposeSemanticHelper.java:303 — The $el_id priority chain (contentDescription > id/testTag > fallback hash) and role mapping are copy-pasted across the XML, accessibility, and Compose paths — three places to edit, already drifting. Extract one shared resolver.

Conventions (per CLAUDE.md)

  • AutocaptureOptions.java:91 — Builder setters (clickOptions/rageClickOptions/deadClickOptions) store @NonNull params with no null guard; a Java caller passing null NPEs later in AutocaptureOptions.isEnabled(), which runs in the MixpanelAPI constructor outside any try/catch — reachable host-app crash during getInstance(). The
    sibling MixpanelOptions.Builder.autocaptureOptions added in this PR does guard; mirror it.
  • AutocaptureManager.java:57,59mRageClickTracker and mDeadClickDetector are assigned only in the constructor but declared non-final ("Final fields for immutability").
  • ClickEvent.java:23 — Instance fields (x, y, elementId, …) omit the m prefix used by every other new class in this PR ("Member variables prefixed with 'm'").

rahul-mixpanel and others added 2 commits July 6, 2026 18:11
Compose screen embeds an XML TextView updated by a Compose button;
XML screen embeds a ComposeView button that updates an XML TextView.
Both reproduce the false dead click from cross-framework UI changes.
ComposeUiChangeMonitor now also monitors the XML view hierarchy so that
a Compose button click which only modifies XML views (e.g. TextView text)
is correctly detected as a UI change instead of a false dead click.

Changes:
- ComposeUiChangeMonitor takes rootView, captures XML content hash baseline,
  checks both Compose semantic tree and XML hash at timeout
- Attach ViewTreeObserver listeners for early cancellation on XML layout/scroll
- Extract computeContentHash as shared method on DeadClickDetector
- Add instrumented test (AutocaptureMixedInstrumentedTest) and test activity
- Remove cross-framework section from demo XmlAutocaptureTestActivity

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Jul 7, 2026

Copy link
Copy Markdown

SDK-27

@rahul-mixpanel
rahul-mixpanel marked this pull request as ready for review July 7, 2026 07:34
rahul-mixpanel and others added 2 commits July 7, 2026 16:08
Reset downTime on ACTION_POINTER_DOWN and ACTION_CANCEL so the
existing downTime > 0 guard rejects the subsequent ACTION_UP.
Without this, two-finger gestures (pinch, rotate) produce a
spurious $mp_click when the first finger lifts.

Applied in both CurtainsHelper and AutocaptureManager fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Passing null to clickOptions(), rageClickOptions(), or deadClickOptions()
would set the backing field to null and NPE on the first isEnabled() call.
Added defensive null checks matching the existing MixpanelOptions.Builder pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread analytics/src/main/java/com/mixpanel/android/autocapture/AutocaptureManager.java Outdated
rahul-mixpanel and others added 16 commits July 7, 2026 16:43
Convert screen coordinates to window-relative coordinates before
matching against Compose's getBoundsInWindow(). In split-screen or
multi-window the window origin is offset from screen (0,0), causing
node lookups to miss or pick the wrong element.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
attachRootViewTouchListener was a best-effort fallback for views without
a PhoneWindow (Toasts, PopupWindows). In practice it captured zero real
interactions: Toasts aren't clickable, and PopupWindow child views
consume touches before reaching the root listener. Removing it
eliminates an untracked listener leak and OnTouchListener clobbering.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Combines countViews() and computeContentHash() into a single snapshotViewTree()
method that captures both view count and content hash in one pass, halving the
number of tree traversals per dead click check.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nodeCount was computed but never used by hasChanged(), which only compares
contentHash. Removing it simplifies computeTreeHash to return a plain int
instead of int[], eliminating per-node array allocations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Tap duration filter: 800ms → 500ms to match iOS and platform long-press
  conventions
- Dead click snapshot: add class name, CompoundButton checked state, and
  enabled state to hash; add alpha > 0 visibility check

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Route all autocapture events through the Autocapture class instead of
direct track() calls. This ensures $mp_autocapture=true is set on all
click events (was missing before) and exposes public methods for host
apps to emit click events using ClickEvent + Builder.

Key changes:
- Add trackClick/trackRageClick/trackDeadClick accepting ClickEvent
- Make ClickEvent, Builder, and toProperties() public for host apps
- Require x, y, elementId in Builder constructor (mandatory fields)
- Add null checks on all public method parameters
- Replace EventEmitter interface with Autocapture reference
- Extract trackAutocaptureEvent/mergeProperties shared helpers
- Update AutocaptureManager and SemanticExtractor call sites

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace squared distance comparison with Math.sqrt() to match the
Euclidean distance calculation used by iOS and JS SDKs. Functionally
equivalent but improves cross-platform code consistency.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add cross-framework button/counter elements to both XML and Compose test
screens for manual testing of dead click detection in mixed UI scenarios:
- XML button → XML text, XML button → Compose text
- Compose button → XML text, Compose button → Compose text

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
8 tests covering all cross-framework button→text combinations on both
XML-based and Compose-based screens:
- XML button → XML text, XML button → Compose text
- Compose button → XML text, Compose button → Compose text

Tests verify that clicks producing UI changes do NOT trigger false
$mp_dead_click events. Test activities updated with mixed-framework
elements (ComposeView in XML, AndroidView in Compose).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace separate XmlUiChangeMonitor and ComposeUiChangeMonitor with a
single UnifiedUiChangeMonitor that auto-detects the UI framework
composition and adapts its detection strategy:

- Pure XML: full structural hash + ViewTreeObserver (unchanged behavior)
- Pure Compose: semantic snapshot only (unchanged behavior)
- Mixed framework: content-only XOR hash for XML + semantic snapshot for
  Compose, covering the complete UI surface without false positives from
  Compose ripple animations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pack view count + hash into a single long to eliminate int[] allocation
  per node during tree walks (~200 allocations per click avoided)
- Replace recursive captureBaseline() fallback with inline XML capture
  to prevent infinite recursion on NoClassDefFoundError
- Use getName() instead of getSimpleName() for class name hashing
  (getName() is JVM-cached, getSimpleName() recomputes each call)
- Cache Compose classpath availability check to skip findComposeRoot
  tree walk entirely for pure XML apps

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, autocapture was initialized unconditionally during SDK init
and continued capturing events even after the user opted out of tracking
at runtime. This violated the opt-out contract.

Changes:
- Guard autocapture init behind hasOptedOutTracking() check so it is
  not started when the user has previously opted out
- Store AutocaptureOptions for later use by optInTracking()
- Stop mAutocaptureManager in optOutTracking() to halt event capture
- Start/recreate mAutocaptureManager in optInTracking() to resume
- Add AutocaptureManager.isStarted() public getter for test access
- Add MixpanelAPI.getAutocaptureManager() package-private for tests

Tests:
- testAutocaptureNotStartedWhenOptedOutByDefault
- testAutocaptureStopsOnOptOut
- testAutocaptureRestartsOnOptIn
- testAutocaptureStartsOnOptInAfterOptOutByDefault

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use platform-agnostic naming instead of web-specific ariaLabel.
The API property name ($attr-aria-label) is unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment on lines +236 to +284
private static ClickEvent.Builder extractFromNode(@NonNull SemanticsNode node, float x, float y) {
SemanticsConfiguration config = node.getConfig();

String contentDesc = getStringProperty(config, SemanticsProperties.INSTANCE.getContentDescription());
String testTag = getStringProperty(config, SemanticsProperties.INSTANCE.getTestTag());

// Element ID resolution (matching Android plan):
// 1. contentDescription (from semantics { contentDescription = ... })
// 2. testTag (from Modifier.testTag(...)) - equivalent to resource ID
// 3. ClassName_view_<hashCode>
String elementId = null;
String accessibleLabel = null;

if (contentDesc != null && !contentDesc.isEmpty()) {
elementId = contentDesc;
accessibleLabel = contentDesc;
}

if (elementId == null && testTag != null && !testTag.isEmpty()) {
elementId = testTag;
}

if (elementId == null) {
String tagName = getTagName(config);
elementId = tagName + "_view_" + Integer.toHexString(node.hashCode());
}

ClickEvent.Builder builder = new ClickEvent.Builder(x, y, elementId);

if (accessibleLabel != null) {
builder.accessibleLabel(accessibleLabel);
}

// Tag name from role
String tagName = getTagName(config);
builder.tagName(tagName);

// Role
String role = getRoleString(config);
builder.role(role);

// Interactive check
builder.isInteractive(isInteractiveElement(config));

MPLog.d(TAG, "Extracted Compose semantics - id: " + elementId +
", tag: " + tagName + ", role: " + role);

return builder;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 $elements hierarchy property missing from Compose events

extractFromNode never calls builder.elements(...), so every Compose-originated $mp_click, $mp_rage_click, and $mp_dead_click event will omit $elements entirely. Both the XML path (extractFromViewbuildHierarchyString) and the accessibility fallback path (SemanticExtractor.extractFromNodebuildHierarchyFromNode) populate this field; the direct Compose SemanticsNode path does not. Any Mixpanel query or funnel that filters on $elements will silently drop all Compose-generated events.

Add clear guidance on what each field represents, how the SDK collects
it, and what values to pass when tracking manually. Align documentation
structure across iOS and Android.

Remove button/clickable text content as elementId fallback in Compose
accessibility node extraction — text on interactive elements should not
be collected as identifiers.
Align Android role values with iOS: PascalCase (Button, Switch, Slider,
etc.) and null when no role is detected instead of "none". Also align
role names: img → Image, textbox → TextField, combobox → ComboBox,
checkbox → Checkbox, radio → Radio.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants